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Abstract 

D espite a robust body of evidence of effectiveness of social 
programs, few evidence-based programs have been scaled 
for population-level improvement in social problems. 

Since 2010 the federal government has invested in 
evidence-based social policy by supporting a number of 
new evidence-based programs and grant initiatives. These initiatives 
prioritize federal funding for intervention or prevention programs 
that have evidence of effectiveness in impact research. The increased 
attention to evidence in funding decision making is promising; 
however, to maximize the potential for positive outcomes for children 
and families, communities need to select programs that fit their needs 
and resources, the programs need to be implemented with quality, and 
communities need ongoing support. Drawing on experiences scaling 
evidence-based programs nationally, the authors raise a number of 
challenges faced by the field to ensure high-quality implementation and 
discuss specific proposals, particularly for the research and university 
communities, for moving the field forward. Recommendations include 
designing and testing intervention and prevention programs with an 
eye towards scaling from the start, increased documentation related to 
implementation of the programs, and working toward an infrastructure 
to support high-quality, effective dissemination of evidence-based 
prevention and intervention programs. 
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From the Editors 


As rigorous evidence grows for various interventions, programs, and initiatives 
to address social problems across various areas, such as family support, 
education, health, and employment, there is a continual need to scale up 
these programs to reach individuals, families, and communities most in 
need of the services. However, there are challenges in the scaling of many 
evidence-based interventions, programs, and initiatives to attain the expected 
results. In this Social Policy Report (SPR), Supplee and Metz raise a number 
of issues, particularly for the research community to support the scale up 
of evidence-based programs (EBPs). In particular they highlight the need for 
infrastructure, including information-rich documentation, to support effective 
dissemination and utility of EBPs. 

Four commentaries expand on the issues raised in the Supplee and Metz 
paper. First, Bumbarger acknowledges the struggle of scaling innovation and 
evidence-based programming, especially in the midst of technology while 
also providing optimism about its potential role in advancing EBPs. Second, 
Domotrovich and Durlak emphasize the importance of infrastructure to 
support new researchers and practitioners focused on scaling of evidence- 
based programming. In the third commentary, Bogard, Karoly, and Brooks- 
Gunn bring to bear the value of considering the cost-benefit as part of the 
equation in determining which EBPs to scale up. Finally, Lopez Boo brings an 
international perspective by highlighting that while Latin American countries 
understand the importance of EBPs, expected effects are limited due to the 
lack of quality improvement data and challenges in maintaining fidelity at 
scale. 

Considering the growing field of scaling evidence-based interventions, 
programs, and initiatives, within the U.S. and internationally, the SPR authors 
and commentators, together, emphasize the need for intentionality and 
proactiveness in considering scaling at all levels from the developer, purveyor, 
practitioner, to the community, as well as addressing the fit and economics 
of EBPs, especially for communities and countries with limited resources and 
trained staff. 


- Iheoma U. Iruka (Issue Editor) 
Samuel L. Odom (Editor) 
Kelly L. Maxwell (Editor) 




Opportunities and Challenges 
in Evidence-based Social Policy 


O ver the past thirty years an explosion 
of research has looked at the impact 
of social programs designed to 
improve outcomes for children and 
families. However, few evidence- 
based programs have been truly 
scaled for population-level impact on social problems 
(lOM and NRC, 2014; Rhoades Cooper, Bumbarger, & 

Moore, 2013; Rotheram-Borus, Swedenman, & Chorpita, 
2012). There has been a growing movement toward the 
use of evidence-based policy, or public policy that directs 
funding toward programs or practices that have evidence 
of achieving outcomes. Evidence-based policy, a concept 
borrowed from the health sciences, involves integrating 
the best scientific knowledge, clinical experience, and 
input from clients in order to choose the best course of 
action for addressing a problem (Sackett, Rosenberg, 

Gray, Haynes, & Richardson, 1996). The interest in 
applying these concepts to social policy comes, in part, 
from increasing pressure from federal, state, and local 
funders to demonstrate improved outcomes for children 
and families from social expenditures (Haskins & Baron, 
2011 ). 

Since 2010 the federal government has invested 
in evidence-based social policy through a number of 
new evidence-based programs and grant initiatives 
(see Table 1, Haskins & Margolis, 2014; for background 
on the initiatives see Haskins & Baron, 2011). Each of 
these initiatives prioritizes funding for intervention 
or prevention programs that demonstrate evidence 
of effectiveness in impact research. For example, as 
directed in the authorizing legislation, the majority of 
the $1.5 billion dollars over 5 years of the Maternal, 

Infant and Early Childhood Home Visiting Program must 
be spent on early childhood home visiting models that 
have demonstrated evidence of effectiveness. A benefit of 
this increased emphasis on evidence- based policy is the 
integration of research evidence in program funding and 
implementation, but one challenge for the field will be 


delivering on the anticipated improved outcomes through 
the use of evidence-based programs. The execution 
of the evidence-based initiatives has highlighted some 
critical gaps that must be filled to successfully support 
the further expansion of evidence-based policy. One 
critical element of evidence-based policy is having 
programs that have rigorous evidence of effectiveness. 
However, if evidence-based programs are not designed 
for scaling, disseminated effectively, or supported for 
quality implementation, we run the risk of not being able 
to deliver on promised outcomes for children and families 
(Haskins a Baron, 2011; lOM and NRC, 2014). 

The purpose of the current Social Policy Report is 
to highlight what we have learned about implementing 
and scaling up evidence-based programs. The literature 
has noted a complex set of stakeholders are necessary 
to scale-up evidence-based programs including 
researchers, funders, communities, practitioners, 
trainers, professional associations, private industry and 
more. While this article cannot address the roles of all of 
these stakeholders, this Social Policy Report will focus on 
the role of the child development research community 
in helping to address this challenge and move the field 
forward. 

Using Evidence to Inform Decision Making 

In discussions related to decision making in policy with 
the scientific community, often there is a belief that 
the research to policy connection is linear— research 
moves from a researcher to policymaker to be applied 
to a specific decision (Nutley, Walter, & Davies, 2007). 
Without being able to point to a specific study (or set of 
studies) that influenced a policy decision, often there is a 
naive belief that evidence is not informing policy. There 
is a growing acknowledgement in the policy community 
that evidence from research is not the only factor in 
decision making in social policy and education and the 
application of evidence to decision making is very non- 


linear and complex (Haskins & Baron, 2011; Nutley et al., 
2007; Tseng & Nutley, 2014). In practice, there are many 
different kinds of data people use as evidence in decision 
making and policy (e.g., administrative data, experience, 
stakeholder input, research; Nutley et al., 2007). For 
practitioners deciding whether to adopt an evidence- 
based program, variables such as the acceptability of 
the program to staff or clients, the readiness of the 
program to be implemented in the communities, and 
the resources necessary to implement are all factors at 
play in addition to the research that shows the program 
is effective (Dworkin, Pinto, Hunter, Rapkin, & Remien, 
2008). Providing decision makers with a broad synthesis 
of evidence as well as information about implementation, 
acceptability, and feasibility are important to the 
evidence-based policy movement. 

There has been a 
rapid increase in federally- 
sponsored systematic reviews 
that support states’ and 
communities’ selection of 
programs or practices with 
evidence of effectiveness 
(see Table 2). Behind most 
systematic reviews are the 
goals of: 1 ) consolidating 
the available evidence on a 
particular topic; 2) assessing 
the quality of the available 
impact evidence; and 3) 
making determinations 
about the state of evidence 
for specific programs or 
practices. Some aim to contribute knowledge about 
evidence to policy and practice broadly (e.g., U.S. 
Department of Education’s What Works Clearinghouse), 
while others have the more high-stakes purpose of 
determining which programs are eligible for funding in 
an evidence-based initiative (e.g., U.S. Department of 
Health and Human Services teen pregnancy prevention 
review). While many systematic reviews focus primarily 
on the impacts of the program, some systematic 
reviews provide implementation information without an 
assessment of a program’s readiness to scale up (e.g.. 
Home Visiting Evidence of Effectiveness), while others 
also provide assessments of implementation readiness 
(e.g.. National Registry of Evidence-Based Programs and 
Practices). However, for both information on impacts and 
implementation, systematic reviews are only as good as 


the information available to review, and there remain 
large gaps in available information on implementation 
(e.g., Paulsell, DelGrosso, & Supplee, 2014). 

Systematic reviews of evidence provide a valuable 
service by summarizing and assessing the quality of the 
full body of available research. They can highlight that 
our knowledge remains limited in many areas (Avellar 
& Paulsell, 201 1 ). For instance, most evidence is from 
small-scale, tightly controlled efficacy trials with limited 
information regarding external validity (Bonell, Oakley, 
Hargreaves, Strange, & Rees, 2006; Green & Glasgow, 
2006). Most fields of interest also have limited available 
replication trials, further restricting the external validity 
of the available evidence (Goesling, Lee, Lugo-Gil, & 
Novak, 2014; Valentine et al., 2011). 

In addition, there is a gap in our knowledge on why 
and how impacts vary by specific 
population groups, settings, and 
other factors, making it hard to 
know with whom to implement 
a program and how to obtain 
the impacts found in efficacy 
trials (Avellar & Paulsell, 

2011; Green & Mercer, 2001). 
When scaling up evidence- 
based programs, implementing 
agencies often want to make 
changes or adaptations to fit 
their populations or settings. 
However, there is little to 
no empirical or theoretical 
identification of the core 
components of evidence-based 
programs to guide these decisions, leaving implementing 
agencies unsure if changes may attenuate impacts 
(Blase & Fixsen, 2013). There is often little information 
available about implementation during the impact trials 
and the features necessary to replicate the program 
tested (Milat, King, Bauman, & Redman, 2011; Paulsell et 
al., 2014; Spoth et al., 2013). 

Though these gaps exist, social policy must move 
ahead implementing programs and serving children and 
families and cannot wait for the perfect complement 
of empirical evidence prior to advancing. Program 
administrators lack information to decide if an evidence- 
based program fits their needs, population, or resources. 
Evidence-based policy emphasizes programs and policy 
that use the best available evidence to support decision 
making while continuing to advance the science. In our 


Providing decision makers with 
a broad synthesis of evidence 
as well as information about 
implementation, acceptability, 
and feasibility are important to the 
evidence-based policy movement. 



experience in scaling evidence-based programs, without 
much of the information discussed above, decision 
makers and implementing agencies are making do with 
what information is available and hoping the promise 
of better outcomes for children and families via an 
evidence-based program will still be attained. 

What We Have Learned About Implementing 
and Scaling Up Evidence-based Programs 

Researchers have identified several key functions and 
structures that need to be developed and installed to 
support the use of evidence in practice (Damschroder 
et al., 2009; Graham et al., 2006; Livet, Courser, & 
Wandersman, 2008; Nutley & Homel, 2006; Rycroft- 
Malone, 2004). However, real world challenges to 
successful implementation still exist. Our experience 
points to some potential strategies to address the 
challenges of scaling up evidence-based programs, 
including: 1) fitting evidence to local context; 2) ensuring 
communication and feedback loops among research, 
policy, and practice; 3) supporting data-driven continuous 
quality improvement; 4) developing organizational 
environments; and 5) clarifying stakeholder roles and 
functions. 


Fitting Evidence to Locai Context 

Despite increasing emphasis on the importance of 
translating, adapting, and optimizing evidence-based 
programs in local contexts, it is not always clear how to 
do this. For example, in a study examining sustainability 
of evidence- based programs after initial implementation, 
a challenge noted by communities is a lack of fit between 
the program and their population, setting, and needs 
(Rhoades Cooper et al., 2013). When many changes 
were needed to shoe-horn a program into a community 
setting, sustainability suffered (Rhoades Cooper et 
al., 2013). Similarly, challenges implementing a teen 
pregnancy prevention program were related, in part, to 
a lack of understanding at the outset of requirements 
to implement the program. For example, organizations 
lacked consistent classroom space or health educators 
had challenges carving out the required number 
of classroom sessions. Both of these are examples 
of disconnects between the needed and available 
infrastructure and resources to implement a program 
(Demby et al., 2014). 

The interplay between the service delivery 
organization, providers, and clients calls for early and 
ongoing assessments of these multi-level characteristics 


to determine whether a good enough fit exists between 
an evidence-based program and local context (Aarons 
et al., 2012). A good fit may require alignment of 
service system-level characteristics including funding, 
policy, and regulations and alignment of organizational 
characteristics that support successful implementation 
including leadership, culture, and climate (Panzano, 
Sweeney, Seffrin, Massatti, & Knudsen, 2012). At the 
provider level, key factors for successful use of evidence- 
based programs include staff attitudes towards evidence 
and the supervision to support implementation for staff 
(Dymnicki, Wandersman, Osher, Grigorescu, & Huang, 
2014). At the client level, characteristics that influence 
successful implementation of evidence-based programs 
include alignment of cultural background of clients and 
program, and client motivation for participation (Dariotis, 
Bumbarger, Duncan, & Greenberg, 2008). Jurisdictions 
seeking to implement evidence-based programs need 
guidance on how to assess these key factors to inform 
decisions to adopt an evidence- based program and/or 
tailor aspects of the system or the intervention. 

Comprehensive needs assessments can support the 
assessment of fit by determining the alignment between 
the needs of the community, the outcomes targeted by 
the evidence-based program, and the resources available 
to support implementation. A needs assessment helps 
communities identify local risk factors that may be 
better addressed by one evidence-based program over 
another. A thorough needs assessment may determine if 
the implementing agency has the resources necessary, 
such as the ability to find and hire health educators for 
a pregnancy prevention program. Key aspects of a robust 
needs assessment to support program selection include: 
selecting and refining the characteristics and needs of 
the target population(s); identifying and confirming 
barriers to care; assessing the level of risk and protective 
factors; establishing a theory of change; examining the 
evidence base; and engaging opinion leaders (Bryson, 
Akin, Blase, McDonald, & Walker, 2014). Our experience 
related to scaling up evidence-based programs has 
highlighted the need for states and communities to get 
support and guidance from the scientific community on 
conducting comprehensive needs assessments using multi- 
method, multi-source iterative methods. Frameworks 
such as Communities that Care (Hawkins et al., 2012) 
and PROSPER (Spoth & Greenberg, 2011) support 
communities to assess fit of evidence-based programs. 
Rigorous research shows impacts from both of these 
frameworks on outcomes for children and families and 


long-term sustainability (Feinberg, Jones, Greenberg, 
Osgood, & Bontempo, 2010; Hawkins et al., 2012; Spoth 
& Greenberg, 2011). Unfortunately, frameworks such as 
these are not widely scaled themselves. 


Ensuring Communication and Feedback Loops 
Between Research, Poiicy, and Practice. 

Frequent and inclusive communication between key 
stakeholders has been established as a factor of 
successful implementation (Hurlburt et al., 2014). Studies 
have found that stakeholders are more likely to persevere 
in the face of implementation challenges when early 
implementation successes are shared between developers 
and implementing agencies, making communication 
regarding the achievement of implementation milestones 
especially important (Aarons et al., 2014; Rhoades 
Cooper et al., 2013). In examining the sustainability of 
quality implementation of evidence- based programs over 
time, in juvenile delinquency programs in Pennsylvania 
(Rhodes Cooper et al., 2013) and mental health 
programs in Ohio (Panzano, Sweeney, Seffrin, Massatti, 

& Knudsen, 2012), a key variable was the quality of the 
relationship between the implementing agencies and the 
evidence- based program. Therefore, the practitioner- 
to-developer communication loop seems to be a key 
aspect of successful efforts to implement evidence-based 
programs. Ongoing communication between developers 
and implementing agencies is particularly critical during 
the early phases of implementation when frequent 
troubleshooting to address challenges is needed. As 
implementation stabilizes, the accountability of key 
functions necessary for effective implementation is 
often shifted to intermediary organizations and local 
implementation teams. 


Supporting Data-driven Continuous Quaiity 
Improvement. 

There is increasing emphasis on the role of continuous 
quality improvement (CQ.I)— the use of administrative 
data to monitor the quality of implementation and 
outcomes and then strategically modify systems or 
services to optimize processes, procedures, and outcomes 
(Kritchevsky & Simmons, 1991). CQ.I has been highlighted 
as a core element of effective implementation because 
it enables ongoing learning among practitioners, program 
administrators, and developers. This dialogue tying data 
monitoring and quality implementation can improve the 
sustainability of evidence in practice settings (Chambers, 
Glasgow, & Stange, 2013). 


CQ.I has implications for practitioners, program 
administrators, and developers. For practitioners, CQ.I 
creates feedback loops among clients, practitioners, and 
administrators that can be used to continually assess and 
improve practice (Chambers, Glasgow, & Stange, 2013). 
For program administrators, CQ.I provides leadership 
teams with frequent information about what is helping 
or hindering efforts to make full and effective use of 
evidence at the practice level, which can result in 
successful systems change (Khatri & Frieden, 2002). The 
information may consist of descriptions of practitioner 
experiences, administrative data, fidelity monitoring 
data, or survey or focus group data. Leadership can 
use information from practitioners to reduce systems 
barriers to implementation (Fixsen, Blase, Metz, & 

Van Dyke, 2013). Similarly, developers can use the 
feedback loops of information produced through CQ.I 
efforts to continuously improve their program and 
share improvements with other implementing agencies. 
Developers can also use evaluation methods such as rapid 
cycle evaluation or clinical trials using administrative 
data to support improvements in the program (e.g., 
Ammerman et al., 2007). 


The Importance of Organizations. 

The capacity of the organization or system may be 
as necessary as the specific evidence- based program 
(Glisson et al., 2012). When there is not a good 
fit between an evidence- based program and an 
implementing agency, too often the programs are 
modified to accommodate current systems structures 
rather than vice versa. Implementing agencies may not 
understand what is needed to create an organizational 
context that can support evidence- based programs. 
Research has identified several characteristics at 
organizational and systems levels that can make or 
break the successful implementation of evidence-based 
programs: networks and communication within an 
organization; the culture and climate of the organization; 
and the organization’s readiness for implementation 
(i.e., leadership engagement, access to information and 
knowledge) (Damschroder et al., 2009; Dariotis et al., 
2008). Developers need to clearly articulate up front 
the technical and organizational resources needed to 
deliver an evidence-based program (Milat, King, Bauman, 
& Redman, 2012). Ideally developers would both design 
programs with organizational limitations and challenges 
in mind, as well as come to understand the aspects of 
organizations that are necessary to effectively implement 
the evidence- based program. 


Agencies need to be aware that changes to the 
status quo may be needed within a system to support 
the evidence-based program. Unfortunately, there is a 
small base of empirical information to draw on related to 
how to change organizations and build their capacity to 
implement an evidence- based program. More research is 
needed to help fill this gap in knowledge. 


Clarifying Stakeholder Roles and Functions. 

Successful uptake of evidence requires deep interaction 
among researchers, service providers, policy makers, 
and other key stakeholders (Flaspohler, Meehan, Maras, 
& Keller, 2012; Palinkas et al., 2011; Wandersman et 
al., 2008). However, key stakeholders (e.g., service 
providers, policy makers, funders, program experts, 
technical assistance providers) 
are often unclear about their 
specific roles in supporting 
implementation. This 
confusion can increase as 
implementation moves from 
early stages of exploration to 
initial implementation stages 
(Aarons et al., 2014; Hurlburt 
et al., 2014). However, there 
are multiple ways program 
implementors can clarify the 
roles and functions of key 
stakeholders, including: get 
buy in for the implementation 
of a specific evidence- 
based program across all 
stakeholder groups; ensure 
that all stakeholders have a 
broad understanding of the 
underlying program logic; 
collaborate and communicate 
frequently with stakeholders throughout all stages of 
implementation; explicitly negotiate stakeholder roles 
and responsibilities; and share data used for continuous 
quality improvement (Metz, Albers, & Mildon, 2015). 
Developers can help by providing clear, codified, and 
coherent logic models for their programs that agencies 
can use to guide implementation activities. In addition, 
developers need to decide how much control over 
implementation they want to have as their program is 
scaled up to new sites, as well as an explicit vision for 
how quality implementation will be sustained. 


What Facilitates or Impedes Effectives 
Scaling of Evidence-based Programs? 

What makes an evidence- based program ready to scale? 
What kinds of information are necessary to scale up 
evidence- based programs? In this section we highlight 
a few critical elements. The first element includes 
designing intervention and prevention programs with the 
end user and context in mind. The second element is 
creating a marketing and distribution system to facilitate 
dissemination of evidence- based social programs. 


Designing Evidence-based Programs With the 
End User and Context in Mind 

Often when federal, state, or local entities have tried 

to scale evidence- based 
programs, they discover 
an invisible infrastructure 
is needed to support high- 
quality implementation. 

To date, developers of 
evidence-based programs 
typically find they are tasked 
with providing the invisible 
infrastructure, in addition 
to all of the support needed 
for scaling their program, 
including consulting with sites 
about program selection, fit, 
adaptation, requirements for 
implementation, providing 
initial and ongoing training 
and technical assistance, and 
support for maintaining fidelity 
to the program (Margolis & 
Roper, 2014). Unfortunately, 
currently there are few 
support systems other than developers to provide these 
services (Fixsen, Blase, Naoom, & Wallace, 2005; lOM and 
NRC, 2014; Sandler et al., 2005). 

Through informal discussions with developers 
of evidence-based programs who have scaled or are 
currently attempting to scale their program, a number 
of pathways to provide this invisible infrastructure 
emerged. Some developers create a business or non- 
profit to provide support services (some leaving academic 
positions to do so), while others try to do two jobs, 
conducting research and supporting implementation. 

Some developers are more interested in research and 


To date, developers of evidence-based 
programs typically find they are tasked 
with providing the invisible infrastructure, 
in addition to all of the support needed 
for scaling their program, including 
consulting with sites about program 
selection, fit, adaptation, requirements 
for implementation, providing initial and 
ongoing training and technical assistance, 
and support for maintaining fidelity to the 
program (Margolis 8^ Roper, 2014). 



development of the program and not as interested in 
supporting implementation, leaving the implementation 
to others who may be more or less familiar with the core 
elements of the program. They may let implementing 
agencies conduct training and oversight themselves, 
develop an organization to fill that role (e.g., Nurse- 
Family Partnership’s National Service Office or the 
Multisystemic Therapy Services office), or leave the 
responsibility to one of the few organizations whose 
role is to support implementation across evidence- 
based programs (e.g., Evidenced-based Prevention and 
Intervention Support Center, http://www.episcenter. 
psu.edu/ or Connecticut Center for Effective Practice, 
http://www.chdi.org/ccep-initiatives.php). Some 
developers prefer a large role in ensuring their program 
is disseminated with fidelity, while others have sold the 
copyright to a company such as a publishing company to 
disseminate on their behalf. These discussions illuminate 
the challenges for developers in understanding issues 
of copyright and ownership, business planning and 
marketing, adult learning for training and technical 
assistance, supporting organizational capacity and 
funding, and many others. 

Many evidence-based programs were designed and 
tested in ideal circumstances. This can create challenges 
related to staffing, engaging and retaining participants, 
as well as other issues. First, challenges arise related to 
staffing, for example, in the ability of the implementers 
(e.g., facilitators, clinicians, teachers) to effectively 
implement the material or challenges in recruiting and 
retaining staff with the required characteristics for the 
program (e.g.. Boiler et al., 2014). Second, challenges 
emerge related to constraints of the settings, including 
policies that limit or prohibit aspects of the programs 
(e.g., state policies that prohibit condom demonstrations 
when that is a core component of a teen pregnancy 
prevention program), limitations on time (e.g., only 
having 12 weeks for a school-based program instead of 
the 14 specified in the program), limitations on material 
resources (e.g., not having enough books at an early 
childhood center to do a literacy activity) (Margolis 
& Roper, 2014; Mattera, Lloyd, Fishman, & Bangser, 

2013). Finally, there may be challenges engaging and 
retaining participants in the real world (Boiler et al., 
2014; Lundquist et al., 2014; Wood, Moore, Clarkwest, & 
Killewald, 2014). The disconnect between the program 
as designed and the actual implementation calls for 
rethinking the way programs are designed and tested, 
and designing them for the circumstances on the ground 
from the start. 


Marketing Distribution System 

In private industry new product development begins with 
a research and development phase. That phase includes 
testing prototypes but also gathering user feedback 
about functionality and appeal. Once the company has a 
product they believe is ready to disseminate, individuals 
who have expertise in marketing and communication 
oversee the product’s distribution. When the product is 
available for purchase, companies employ specialized 
sales staff to make contact with potential users. Finally, 
the company likely has a specialized unit to provide 
ongoing help to users with activities such as installing the 
product or trouble-shooting. 

Currently, in most instances, the development, 
dissemination, and implementation at scale of evidence- 
based programs aimed at improving outcomes for children 
and families is executed quite differently. Often, drawing 
from past research and theory, a researcher develops 
a program and secures funding to test the program in 
a small-scale efficacy trial. This trial often, though not 
always, accesses university support such as graduate 
students or may use a local school district that has a 
relationship with the university. The results of the trial 
are disseminated through peer-reviewed journals that 
only provide enough space for authors to report a few 
analyses of impacts. If successful, the developer may 
engage in a few more impact trials before deciding to 
disseminate the now evidence- based program more 
widely. Alternatively, community agencies may hear about 
the evidence- based program and approach the developer 
about implementing it. The developer at this point is 
faced with creating training manuals and documentation, 
fidelity tools and credential standards for implementing 
staff, answering questions from the implementing sites 
about fit and adaptation, and providing ongoing support. 

In our conversations over the last few years, some 
developers of evidence- based programs mentioned a deep 
desire for more support and guidance in the process of 
dissemination and scale-up. However, others have argued 
that because of the distinct skill set required to market 
and distribute an evidence-based program, the developers 
of evidence-based programs cannot and should not be 
asked to play all of these roles (lOM and NRC, 2014). 
Individuals and organizations with expertise in marketing 
and business should step in to execute the dissemination 
and support of the evidence-based program (Kreuter & 
Bernhardt, 2009). 

Addressing this contrast between the dissemination 
of evidence-based programs and the marketing and 


distribution system in private industry, Kreuter and 
Bernhardt (2009) call for creating a marketing and 
distribution system for public health. Recently, Kreuter 
(2014) has elaborated by calling for a more business- 
like model to disseminate evidence-based programs. He 
asserts the process of developing the program, engaging 
in user feedback, research on market potential and 
feasibility, advertising, and the other steps modeled 
on the private sector approach to scaling innovation 
would lead to a “menu of evidence-based, high-demand, 
practice-ready interventions.” Others have also noted 
this gap in the field and have called for elements 
including feasibility analysis, consumer input, and market 
analysis into production and dissemination of evidence- 
based programs and practices (Rotheram-Borus & Duan, 
2003; Sandler et al., 2005; Spoth et al., 2013). 

An alternate framework for dissemination of 
evidence-based practices comes from the work of 
Rotheram-Borus and Chorpita (Chorpita, Bernstein, & 
Daleiden, 2011; Chorpita 
& Weisz, 2009; Rotheram- 
Borus, Flannery, Rice, & 

Lester, 2005; Rotheram-Borus 
et al., 2012). Rather than 
solely focusing on packaged 
evidence-based programs, 

Chorpita and colleagues argue 
the intervention should be 
looked at in a new frame. 

Specifically, Chorpita and 
colleagues have empirically 
identified the core elements or 
components of mental health 
treatments across evidence- 
based programs. Using these 
elements, they have created a 
system to allow empirically-driven guidance for clinicians 
to serve the children who don’t neatly fit into the narrow 
criteria of many evidence-based treatments (Chorpita et 
al., 2011). Separately, Rotheram-Borus has put forward 
the idea of addressing many of the public’s needs for 
support in easy-to-access formats (e.g., social media, 
using paraprofessionals, placing services in public spaces 
like malls) that can reach the majority of the population 
that may not need a specific targeted program but could 
benefit from just-in-time support (Rotheram-Borus et al., 
2005; Rotheram-Borus et al., 2012). To accomplish this 
aim, Rotheram-Borus and colleagues emphasize that the 
characteristics that attract customers and brand loyalty 


such as attractiveness, accessibility, utilization, and 
demand are requirements for evidence-based programs. 
However, at this time there is little empirical support for 
this concept; more research is necessary to know if this is 
a fruitful avenue to pursue. 

Conclusion 

The investment in evidence-based policy is an exciting 
trajectory for social policy, full of potential for improving 
outcomes for children and families. If we want to see 
this investment pay off, though, we need to make sure 
to put in the hard work it takes to implement and scale 
up evidence-based programs. Our work on current 
evidence- based initiatives has highlighted what this 
entails. First, intervention or prevention programs must 
be designed from the start to be implemented at scale, 
requiring more regular feedback between developers, 
program administrators, and potential future clients of 

the program. Second, both 
reporting more detailed 
information on implementation 
during the trial and 
documenting necessary 
elements for implementation 
when replicating the program 
are essential. Finally, the field, 
including research, policy, 
and practice, needs to think 
critically about the roles and 
responsibilities of all of the 
stakeholders in the system and 
what supports or infrastructure 
are necessary for scaling and 
dissemination. 

What is the Role for the Research Community 
and Universities to Address This Challenge? 

The process of scaling up evidence-based programs 
involves multiple stakeholders (e.g., funders, 
communities, researchers, purveyors) with multiple 
roles in the dissemination system, all critical to the 
success of evidence-based social policy. Though only 
two stakeholders of a much larger system, the article 
concludes with some specific recommendations for child 
development researchers and the university system to 
play in this process. 


If we want to see this investment 
pay off, though, we need to make 
sure to put in the hard work it 
takes to implement and scale up 
evidence-based programs. 



The role of child development researchers. 

Building partnerships with the practice community, in- 
cluding government, communities, schools, or non-profits 
is an important role for researchers. This interaction 
is not always easy but the payoff can be great. Effec- 
tive partnerships can help the issues of fit of evidence 
to local context, communication and feedback loops 
between stakeholders, and the use of data to support 
quality implementation. Models like the practice-based 
research networks (Green & Mercer, 2001; Westfall, Mold, 
& Fanagan, 2007) have been fruitful in building research 
agendas that are responsive to both practice and re- 
search. Research is more likely to be used by the practice 
community because of this engagement (Mold & Peterson, 
2005). In addition, there is a clear need for the practice 
community to collect meaningful data to monitor the 
needs of children and families, to help identify evidence- 
based programs that will address those needs, and to 
monitor the quality of implementation and outcomes over 
time. Many in the practice community need support from 
researchers to choose measures, develop measureable 
goals, build data systems to collect and analyze the data, 
and apply the data to decision making and needs assess- 
ments. If scaled more broadly, systems like Communities 
that Care, PROSPER, and Getting to Outcomes can sup- 
port this process. 

To support assessing fit of evidence to a 
context, researchers and developers need to think 
about scalability at the design phase (Milat et al., 

2012). In addition, researchers/developers must 
consider the elements necessary along the lifecycle 
of implementation. A prerequisite to scaling is having 
a program that is standardized and can be replicated 
with fidelity (Carttar, 2014); therefore, it is critical that 
developers document procedures and activities from 
design through execution and revision. One possibility 
is to begin maximizing the more flexible space with 
online appendices to address space limitations in printed 
manuscripts. It is important for researchers to pair with 
the probable users of this program from the start. This 
means conducting user-input and market analysis along 
with understanding the resources and constraints of 
the setting for which the program is designed. Finally, 
empirically identifying and testing core components of 
the evidence-based program are essential to supporting 
quality implementation as states or communities make 
decisions about appropriate adaptations (Blase & 

Fixsen, 2013). This may mean considering a Multiphase 
Optimization Strategy (MOST) framework (Collins, 


Murphy, & Stretcher, 2007) or, at a minimum, testing 
mediators and moderators, dropping elements not 
strongly tied to impacts. 

The field of implementation science has been 
rapidly expanding and building knowledge. At this point, 
however, more questions than answers remain. Data 
on implementation is critical for understanding and 
supporting the potential for scaling. The research and 
practice community need to jointly develop a better 
understanding of implementation support systems. 
Support systems can help with fit, communication 
and feedback loops, and clarifying roles and function 
of stakeholders. Building organizational or system 
capacity to adopt and sustain evidence-based programs 
continues to be necessary. We have promising evidence 
of the ability to support practitioners to choose an 
evidence-based program and effectively sustain it from 
the frameworks cited throughout this article. More 
research related to building the capacity of the practice 
community to have the resources and infrastructure 
necessary to implement an evidence-based are 
particularly needed, for example, effective training and 
coaching models for staff development. 


The role of universities 

We see three roles for universities specifically. First, the 
students served by universities are the future program 
administrators who will be selecting and implementing 
evidence-based programs. These future staff need the 
skills to understand emerging evidence, how and why 
evidence-based programs should be implemented in 
certain ways (i.e., assessing fit), and how to interpret 
data used in needs assessments and implementation 
monitoring (i.e., CQ.I). Universities touch bachelors, 
masters, and doctoral level students, all of whom will 
likely have a role in this larger service system. Innovative 
ideas such as universities that are exploring the option of 
offering a set of courses specifically related to evidence- 
based social policy are emerging but at this point 
remain unique in the larger university system [e.g.. The 
Pennsylvania State University is working on creating such 
a program (Small, 2014) and University of Washington 
School of Social Work’s certificate in prevention science 
(http: / /socialwork. uw. edu)] . 

A second role for universities is providing the 
infrastructure and support for academic researchers to 
market and disseminate evidence-based programs they 
develop. A promising approach may be partnerships 
between developers, often social science or public health 


faculty, and business and marketing faculty (Rotheram- 
Borus et al., 2012). Specifically, business colleagues may 
provide expertise on negotiating business agreements, 
obtaining copyright, negotiating conflicts of interest, 
and small business development. Another approach is a 
technology transfer office such as one at the University 
of Illinois that provides intensive supports for faculty in 
the distribution of education programs (see http://otm. 
illinois.edu/). 

Finally, universities could provide the infrastructure 
for university-based implementation support centers 
that simultaneously empirically study implementation 
and support quality implementation and sustainability. 
Centers like the Evidence- based Prevention and 
Intervention Support Center at The Pennsylvania State 
University (EPIS Center) or The Center for Communities 
That Care at the University of Washington are uniquely 
situated both to provide elements of the invisible 
infrastructure and to empirically study dissemination and 
scale-up to contribute to the larger knowledge base. As 
research in the field of implementation and dissemination 
frequently requires close collaboration between research 
and practice communities, support from universities 
for these partnerships and the empirical study of these 
domains is critical to advance the field. 

For a more in-depth discussion of some of these 
topics see Spoth et al. (2013), Haskins and Margolis 
(2015), and lOM and NRC (2014). ■ 
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Table 1 (Haskins & Margolis, 2014, p. 

Tiered Evidence Initiatives 

135) 



Administering Agency 

Funding 

Maternal, Infant and Early Childhood Home 
Visiting 

Department of Health and Human Services 

$1 .5 billion in Patient Protection and Afford- 
able Care Act of 2010 

Teen Pregnancy Prevention 

Department of Health and Human Services 

$110 million in Consolidated Appropriations 

Act of 2010 

Social Innovation Fund 

Corporation for National and Community 
Service 

$50 million in the Consolidated Appropriations 
Act of 2010 

Investing in Innovation 

Department of Education 

$650 million in the American Recovery and 
Reinvestment Act of 2009 

Workforce Innovation Fund 

Department of Labor 

$125 million in the Department of Defense and 
Full-Year Continuing Appropriations Act of 201 1 
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Table 2 

Federally Sponsored Evidence Reviews 


Name of the Review 

Review Topic 

Agency 

Website 

Evidence-Based Prevention 

Program 

Health programs in chronic dis- 
ease self-management, physical 
activity, diabetes, nutrition, smok- 
ing cessation, fall prevention, and 
medication management 

Administration on Aging (HHS) 

http: / / WWW. healthyagingpro- 
grams.org 

Evidence-Based Practice Centers 

All relevant scientific literature 
on a wide spectrum of clinical and 
health services topics 

Agency on Healthcare Research 
and Quality (HHS) 

http://www.ahrq.gov/research/ 
findings/evidence-based-reports/ 
overview/index, html 

Learning about Infant and Toddler 
Early Education Services 

Out-of-home early care and 
education models for infants and 
toddlers (from birth to age 3) 

The Office of the Assistant Secre- 
tary for Planning and Evaluation 
(HHS) 

No website yet 

Community Guide 

Population-level health interven- 
tions 

Centers for Disease Control and 
Prevention (HHS) 

http:/ /WWW. thecommunityguide. 
org/ 

Teen Pregnancy Prevention 

Evidence Review 

Programs aimed to reduce teen 
pregnancy, sexually transmitted 
infections, and associated sexual 
risk behaviors. 

The Office of the Assistant Secre- 
tary for Planning and Evaluation 
(HHS) 

http://tppevidencereview.aspe. 

hhs.gov/ 

Prevention Research Synthesis 

HIV prevention 

Centers for Disease Control and 
Prevention (HHS) 

http://www.cdc.gov/hiv/dhap/ 

prb/prs/index.html 

What Works in Reentry 
Clearinghouse 

Reentry programs and practices 

National Institute of Justice (DoJ) 

http://whatworks.csgjusticecen- 

ter.org/ 

Clearinghouse for Labor Evalua- 
tion and Research 

Research on labor topics 

Chief Evaluation Office (DoL) 

http://clear.dol.gov/ 

What Works Clearinghouse 

Programs, practices and policies in 
education 

Institute of Education Sciences 
(ED) 

http://ies.ed.gov/ncee/wwc/ 

Employment Strategies for 
Low-Income Adults Review 

Employment and training pro- 
grams and strategies for low- 
income individuals 

Administration for Children and 
Families (HHS) 

http://www.acf.hhs.gov/pro- 

grams/opre/research/project/ 

employment-and-training-evi- 

dence-review 

Home Visiting Evidence of 
Effectiveness 

Home visiting program models 
that target families with pregnant 
women and children from birth to 
age 5 

Administration for Children and 
Families (HHS) 

http://homvee.acf.hhs.gov/ 

Strengthening Families Evidence 
Review 

Strategies to strengthen families, 
including those that encourage 
fathers’ involvement in their chil- 
dren’s lives and support couples’ 
relationships 

Administration for Children and 
Families (HHS) 

http://familyreview.acf.hhs.gov/ 

Crime Solutions 

Criminal justice, juvenile justice, 
and crime victim services 

Office of Justice Programs (DoJ) 

http://www.crimesolutions.gov/ 

National Registry of Evidence- 
Based Programs and Practices 

Substance abuse and mental 
health interventions 

Substance Abuse and Mental 

Health Agency (HHS) 

http: / / WWW. nrepp. samhsa. gov/ 


The opinions expressed in this paper are those of the authors alone and do not represent those of the Administration for Children and Families or Department of Health and Human 
Services. Correspondence should be directed to: Lauren H. Supplee, Office of Planning, Research and Evaluation, Administration for Children and Families, 370 L’Enfant Promenade SW 7th 
FI West, Washington DC 20447; lauren.supplee@acf.hhs.gov 
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Commentary 


Brian K. Bumbarger 
Prevention Research Center, 
Penn State University 


S upplee and Metz 
describe both 
challenges and 
opportunities for 
advancing the use of 
research evidence 
in social policy, and I will expand 
here briefly on three aspects: 
collaboration, infrastructure, and 
technology for big data. 

The concept of evidence- 
based social policy seems so obvious 
and intuitive as to be inarguable, 
and the logical end-point of 
child development research. The 
mission of SRCD is to “achieve a 
comprehensive understanding of 
human development and to foster 
the effective application of that 
understanding to improve human 
well-being”T In other words, the 
Society seeks to continually increase 
our knowledge of child development 
(through rigorous research) and 
to encourage the application of 
this research-informed knowledge 
base to improve child, family, and 
community well-being across diverse 
contexts. 


'SRCD draft strategic plan, retrieved December 19, 

2014 from http://www.srcd.org/about-us/strategic-plan. 


Presented in the form 
of a one-sentence mission 
statement, this research-to- 
policy framework sounds rather 
simple. But we only need to look 
at a typical etiological regression 
model, or to Bronfenbrenner’s 
developmental-ecological model, 
to remind ourselves how complex 
human development can be-and 
subsequently how challenging it 
might then be to craft effective 
social policy, even with the aid 
of strong science. Although it is 
reinforcing (and perhaps comforting) 
to think our 21st-century research 
juggernaut can generate sound 
scientific breakthrough discoveries 
that can readily translate to 
important changes in social policy, 
we know that this isn’t typically 
the default. There is plenty of good 
science that doesn’t make the 
policy/ practice leap; there is plenty 
of social policy that isn’t grounded 
in any science whatsoever; and just 
as importantly, there are plenty of 
timely and important social policy 
questions for which there is no 
informative science. 

One important factor in 
this recalcitrant research-policy 
gap is the disconnect between 


stakeholder groups. Beyond the 
children and families whose lives we 
seek to improve, the other major 
stakeholder groups involved in this 
enterprise include researchers, 
practitioners, and policymakers. 
Researchers, practitioners, and 
policymakers are in the same arena, 
but not often on the same team. 
Although one could argue these 
groups are all seeking to improve the 
human condition, they each clearly 
have different goals, priorities 
and perspectives, and reward 
structures; and true collaboration 
between the three is rare enough 
to be exemplified when it occurs 
(Bumbarger & Campbell, 2012). To 
address this challenge, there is a 
growing call for the development of 
intermediaries (Franks, 2010; Pew- 
MacArthur Results First Initiative, 
2014; Rhoades, Bumbarger, & Moore, 
2012) and backbone organizations 
(Turner, Merchant, Kania, & 

Martin, 2012) to serve as the 
“infrastructure” for moving science 
to practice, bridging across systems 
and silos for collective impact. 

Intermediaries and backbone 
organizations are just one aspect 
of an infrastructure necessary for 
moving important research to policy 



at scale (lOM and NRC, 2014). Every 
new discovery in our understanding 
of child /human development or 
demonstration of the efficacy of an 
intervention can be viewed as an 
innovation. But not every important 
innovation achieves scale. Consider 
for example why smart phones are 
nearly universal, while alternative- 
fuel cars are still far from 
mainstream. While each innovation 
may represent an intellectual or 
technological milestone, there 
is often some further product 
refinement and significant supporting 
infrastructure or capacity required 
for the new “technology” to become 
practically useful or scalable. 
Presently there is no de-facto 
system or structure for moving 
innovations in social/developmental 
science to scale. Federal agencies 
fund research projects that may 
result in new understanding of an 
important etiological mechanism or 
demonstrate the effectiveness of 
a program, but there is no formal 
subsequent mechanism for ensuring 
that the best scientific discoveries 
are then optimized through further 
product development, packaged, 
and distributed through efficient and 
cost-effective channels. 

Finally, we must consider 
the impact of technological 
advances on the entire endeavor 
of child development research 
and social policy. Good science 
is characteristically slow and 
incremental. And perhaps more 
so than at any point in human 
history, the slow methodical pace 
of rigorous science is at odds with 
our rapidly changing world. This is 
not simply a cultural incongruence, 
but a fundamental challenge to the 
validity and practical usefulness of 
our science. Advances in technology 
have impacted every aspect of 


human development, from neurons 
to neighborhoods, cellular to 
societal; and the pace of that change 
is increasing exponentially. Consider 
for example, that through advances 
in nanotechnology more data has 
been created in the past three years 
than the previous 40,000 years (Kim, 
Trimi, & Chung, 2014). To what 
extent has technology changed the 
human experience so dramatically 
that the most fundamental tenets 
of our understanding of child 
development, family dynamics, or 
social determinants of health must 
be re-examined? What impact might 
these changes then have on the 
efficacy of our best interventions? 
While acknowledging this reality 
might seem disheartening or 
overwhelming, it is a valid and 
logical question, and speaks to the 
need for continued investment in 
basic etiological research even while 
we turn our attention and resources 
to translation and dissemination 
research. There is also equal cause 
for optimism about the impact of 
technological advances on child 
development research and the social 
policy. With an emerging focus on 
the use of “big data” to inform 
solutions to wicked social problems 
(Margolis et al., 2014), conduct 
real-time data collection (Stone, 
Shiftman, Atienza, & Nebeling, 

2007), and data analytics to more 
quickly and effectively identify 
emerging trends and monitor the 
impact and cost-effectiveness of 
social programs and policy (Mrazek, 
Biglan, & Hawkins, 2007), the same 
technological advances that raise 
new questions for child development 
might also provide breakthrough 
solutions. 
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Commentary 
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Joseph A. Duriak 
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S upplee and Metz 
(2015) raise many 
excellent and 
thought-provoking 
points in their 
SRCD report on 

evidence-based social policy. In our 
commentary, we focus primarily on 
the implementation of evidence- 
based programs (EBPs). In our 
opinion, the most important factor 
that will ultimately influence the 
wide scale use of EBPs is not how 
many EBPs are developed or how 
strong the research evidence is 
for these programs but whether 
high quality and efficiently 
delivered professional development 
services are available to help local 
practitioners implement programs 
effectively. 

By professional development 
services we mean training of 
implementors that takes place 
before the program is started 
and on-going technical assistance 
(e.g., coaching, consultation) 
once the program begins. There is 
considerable evidence that these 
services are essential for successful 
program implementation when they 
are provided by individuals with the 
necessary theoretical knowledge 


and practical experience (Durlak & 
DuPre, 2008; Pas & Newman, 2013). 
Unfortunately, Supplee and Metz are 
correct when they point out that a 
sufficient infrastructure offering such 
services does not currently exist. So, 
the critical question for the field is 
how can we ensure that sufficient 
resources are available to bridge 
the gap between evidence- based 
research and everyday practice? 

Because the wide-scale use 
of EBPs is an interdisciplinary 
undertaking, we believe two things 
must eventually happen. One is 
that there must be consistent and 
sufficient financial support from 
federal and state agencies in such 
disciplines as public health, mental 
health, and education, to name 
the major settings where EBPs 
are conducted. These agencies 
should fund the infrastructure 
(let’s call them Research and 
Practice Centers, RPCs) devoted 
to research and practice in scaling 
up and implementing EBPs. The 
second thing that needs to occur 
is the development of a trained 
workforce committed to EBPs and 
their effective implementation to 
staff these centers. In effect, we are 
arguing for a system that supports 


new careers devoted to the wide 
scale use of EBPs. In our opinion, 
unless these two things occur, the 
wide scale application of EBPs will 
continue to occur in a piecemeal and 
limited fashion. 

There are models of this 
concept in the literature. For 
example, an Implementation 
Research Institute began operating 
at Washington University in St. 

Louis in 2009 and has been training 
up to 30 professional staff a year 
drawn from around the country in 
implementation science (Proctor 
et al., 2013). The graduates of this 
institute have been very active 
in securing grant funding and 
initiating implementation projects. 
Unfortunately, the funding for this 
institute, which came from the 
National Institute of Mental Health 
and Veteran Affairs, has ended. This 
is why we emphasize that stable and 
continual funding must be available 
if the spread of EBPs is going to be 
taken seriously. 

A good example of a national 
effort comes from the United 
Kingdom. In 2008 this country 
funded centers to translate effective 
research into practice (Baker et al., 
2009). Supported by the National 



Institute of Health Research, nine 
centers were established throughout 
the UK. These centers are based 
at universities and create working 
relationships with local health care 
agencies. This UK example aligns 
with Supplee and Metz’s suggestion 
that university-centered work would 
be an excellent option for scaling up 
EBPs. 

We certainly agree with 
Supplee and Metz that high quality 
implementation is one of several 
essential ingredients for successful 
evidence-based social policy and 
we focus our remaining comments 
on this issue. The authors note 
that only 2 of the 14 websites they 
examined provided information 
about implementation. Such 
information should be a routine 
part of the criteria for any program 
being listed on a site. As Supplee and 
Metz indicate, potential program 
adopters need practical information 
on what resources are needed to 
conduct a program successfully. 

The Collaborative for Social and 
Emotional learning (CASEL) went one 
step further when it conducted its 
review of evidence- based social and 
emotional learning programs (CASEL, 
2013). CASEL only listed programs 
for which professional development 
services were available (www.casel. 
org). This was done based on the 
belief that it was not helpful to 
tell potential adopters that some 
programs were effective unless the 
necessary resources were available 
to help them implement the 
program effectively. The provision of 
professional development services is 
so essential to implementation that 
we would argue federally-sponsored 
clearinghouses or websites should 
not include any programs that do 
not have the capacity to provide this 
service. 


In the context of discussing 
fitting evidence to local contexts, 
Supplee and Metz point out that 
there are factors at multiple levels 
that influence the implementation 
process that need to be taken 
into account. Several conceptual 
models have been developed to 
describe these factors (Berkel, 
Mauricio, Schoenfelder, & Sandler, 
2011; Domitrovich et al., 2008; 
Durlak & DuPre, 2008) and research 
empirically linking these factors 
to implementation outcomes has 
expanded significantly in the last 10 
years (O’Donnell, 2008). If federal 
agencies sponsoring implementation 
adopted a common framework 
to guide their measurement 
requirements for programs, then the 
number of studies with comparable 
measures of implementation 
predictors (e.g., individual and 
organizational factors) and outcomes 
would increase. This would create 
an opportunity for meta-analyses 
examining implementation in a more 
sophisticated way than has been 
possible to date. 

When it comes to studying the 
implementation process, we would 
focus on four critical components: 
fidelity (adherence to the original 
program), dosage (how much of 
the program is delivered), quality 
of delivery (how well different 
program features are conducted), 
and adaptation (what changes are 
made to the original program). As 
noted by the authors in this report, 
adaptations frequently occur when 
programs are introduced into new 
settings, and we need careful 
documentation of what exactly has 
been changed and how any changes 
affect program outcomes. 

By giving due attention to 
implementation federal agencies 
can play an active role in addressing 


another issue that serves as a 
potential barrier to the success 
of evidence-based social policy. 

One way to do this is to make 
sure that grants include resources 
for implementation support. This 
includes the time it takes to achieve 
high levels of implementation, 
the cost of providing training and 
ongoing support to providers and for 
communication between providers 
and program developers, and the 
time for providers to participate in 
professional development. Additional 
grant funding is also needed for 
randomized trials of interventions 
designed to create implementation 
readiness and capacity in individuals 
and organizations, and trials that 
compare different models of 
implementation support. Supplee 
and Metz point out the need for 
empirical research on training and 
coaching. This type of research 
is a precursor for going to scale 
given the fact that the time and 
money typically available in 
community settings for professional 
development rarely equals what is 
devoted to it in research studies. 

We need to know how training and 
support for implementation can 
be delivered most efficiently and 
effectively. The use of technological 
resources (e.g., web-based systems, 
virtual reality simulations) for 
training, feedback, and collaborative 
problem-solving might eventually 
become an important feature of 
future professional development 
services. 

In conclusion, we are 
optimistic about the potential 
power of evidence-based social 
policy, but believe that the success 
of this approach is dependent on 
a collaboration among federal 
agencies, researchers, and 
practitioners that recognizes the 


fundamental importance of not only 
quality implementation, but also 
of what is needed to achieve such 
implementation. With adequate 
and stable financial support that 
creates a viable infrastructure and 
a dedicated workforce, we can 
learn more about how best to go to 
scale with EBPs. As this information 
is generated, federal agencies can 
leverage the systems they have 
already created to disseminate 
new findings and influence future 
research and practice. 
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I t is a pleasure to write a 
commentary based on the 
comprehensive policy article 
on evidence- based policy 
programs for children, 
youth, and families. Supplee 
and Metz make a strong case for 
using research evidence for program 
decisions at the local, state, and 
federal levels. The emphasis on 
using social science evidence for 
choosing programs to implement 
has gained momentum over the 
past two decades. Examples include 
the sets of randomized control 
trials conducted by the Institute 
of Educational Sciences in the past 
several administrations as well as 
the reliance on evidence in the 
Office of Management and Budget 
deliberations during the Obama 
administration (the new book by 
Haskins and Margolis published by 
The Brookings Institution entitled 
Show me the evidence: Obama’s 
fight for rigor and evidence in social 
policy is a fascinating read of this 
history). The Coalition of Evidence- 
Based Policy, directed by Jon Baron, 
and the Washington State Institute 
for Public Policy (WSIPP), directed 
by Steve Aos, are examples of efforts 
being made to summarize the extant 


evidence on a variety of programs in 
the hope that their syntheses will be 
used in decisions about funding (or 
defunding) programs. 

We would like to add that in 
thinking about scaling programs and 
services for children, youth, and 
families the research community 
should also consider economic 
evidence as part of their research 
programs in order to inform funding 
decisions. The Board on Children, 
Youth and Families of the Institute 
of Medicine and National Research 
Council (lOM/NRC) assembled a 
planning committee of experts to 
design a workshop on the use of 
benefit-cost analyses as part of the 
evaluation of prevention programs. 
The workshop was held in late 2013, 
and a summary has been published 
(lOM and NRC, 2014). 

Workshops of the lOM/ 

NRC do not result in specific 
recommendations, unlike consensus 
studies. Rather the purpose is 
to highlight issues that might be 
important to consider on the specific 
topic under discussion. The first 
author of this commentary is the 
director of the lOM/NRC Board on 
Children, Youth, and Families; the 
second author was a member of the 


workshop planning committee; and 
the third author was the chair of the 
workshop planning committee. 

Some of the issues considered 
in the workshop included: 

• What level of research rigor 
should be met before results from an 
evaluation are used to estimate or 
predict outcomes in a cost-benefit 
analyses? 

• What are best practices 
and methodologies for costing 
prevention interventions, including 
the assessment of full economic/ 
opportunity costs? 

• What processes and 
methodologies should be used when 
theoretically and empirically linking 
prevention outcomes to avoided 
costs or increased revenues? 

• Over what time period 
should the economic benefits 
of prevention interventions be 
projected? 

• What issues arise when the 
results of benefit-cost analyses are 
applied to prevention efforts at 
scale? 

• Do benefit-cost results from 
efficacy trials need to be adjusted 
when prevention is taken to scale? 

• Can we define standards that 
all studies should meet before they 



can be used to inform policy and 
budget decisions? 

• How could research be 
used to create policy models that 
can help inform policy and budget 
decisions, analogous to the benefit- 
cost model developed by the 
Washington State Institute of Public 
Policy? 

According to Supplee and 
Metz, the research community 
should include at least three specific 
elements in designing programs that 
can be scaled: interaction among 
multiple stakeholders, reporting 
detailed information on how the 
programs are implemented so that 
they can be replicated and scaled, 
and the need for understanding 
the results of the studies. Speakers 
at the lOM/NRC workshop also 
identified building political will, 
assessing and documenting costs and 
benefits of programs, and reporting 
results in a way that decision makers 
and implementers can understand 
and take action. In essence, 
the workshop brought together 
researchers and decision makers to 
highlight issues to be considered in 
scaling preventive interventions for 
children, youth, and families. 

Two specific approaches. 
Communities that Care (CtC) and 
WSIPP, were presented at the 
workshop to address the call for 
including multiple stakeholders in 
designing and reporting on outcomes 
of programs designed to benefit 
children, youth, and families. In 
etc, Margaret Kuklinski described 
their approach to including multiple 
stakeholders in designing studies 
and evaluations and reporting on 
outcomes of programs designed 
to benefit youth and families. CtC 
is a coalition-driven approach to 
preventive intervention that involves 
mayors, teachers, and parents. The 


coalition of members uses survey 
data from the community to make 
decisions about program selection. 
The coalition then monitors and 
evaluates outcomes to determine 
impact and guide any necessary 
course corrections in programming. 
Having decision makers at the table 
in the design phase is an important 
component of this approach. 

Steve Aos described WSIPP as 
a model that presents policy options 
to legislators in a standardized 
way which allows them to compare 
apples to apples of program benefits 
and costs with follow-up discussions 
to translate some of the more 
difficult concepts such as risk and 
uncertainty. This is another example 
of a process whereby community 
or state level data on program 
implementation— both impacts 
and costs-are used to inform 
decisions about program funding 
and implementation. Aos indicated 
that it is very important to use local 
data that represent local conditions 
and to update reports to decision 
makers with new data annually. Both 
speakers indicated that building a 
coalition or engaging policymakers 
in discussions about design and 
reporting builds political will to scale 
and sustain programs. 

Workshop participants called 
out the importance of collecting 
more information about the key 
elements of program design and 
implementation and linking the 
science on effectiveness with 
funding decisions for programs 
and services. Panelists noted that 
even though cost analyses are 
necessary to identify the resources 
needed to implement programs 
at scale, this type of information 
typically takes a back seat to 
effectiveness and benefit analyses. 

An ingredients-based approach 


to the collection of detailed 
information on the resources used 
for program implementation serves 
to both document the costs of 
program delivery (including required 
infrastructure) and to provide the 
information needed for taking 
programs to scale in a sustainable 
way. In this way, speakers made a 
call to the research community to 
provide rigorous cost analyses, in 
addition to benefit and effectiveness 
analyses, in order to provide a full 
and rigorous assessment of the costs 
to adopt, implement, and sustain 
programs that can in turn inform 
policy decisions. 

In addition, panelists discussed 
how to translate results to inform 
policy and practice. Several types of 
decision makers were identified who 
need information on benefit-cost 
and evidence-based approaches, 
including program specialists who 
write regulations, implement 
programs, and monitor progress 
within the executive branch of 
government. In communicating with 
decision makers, simple, evidence- 
based presentations are needed 
which can convey the strength of 
the available evidence while also 
acknowledging areas of uncertainty. 
Another point raised among panelists 
was the importance of having 
evaluation and implementation 
program staff spend more time 
discussing the program and how it is 
delivered with decision makers. 

Miscommunication between 
researchers and research consumers 
can stall or divert efforts to drive 
funding decisions. To address this 
situation, Jens Ludwig suggested 
that the research community set 
a bar on quality standards that all 
evaluations would meet in order 
to inform policy. Also, identifying 
mistakes made by research 


consumers in interpreting the 
science can inform how to better 
communicate research findings. 

Engaging research consumers in this 
process would be welcome. 

In sum, the lOM/NRC workshop 
highlighted the powerful potential 
that using economic evidence to 
inform investments in children, 
youth, and families could have on 
improving well-being. However, 
all of these efforts must meet 
standards developed by the research 
community. 
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Commentary 


Florencia Lopez Boo 
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D espite years of 
advocacy for 
evidence-based 
early childhood 
development 
(ECD) programs in 
Latin-America (LAC), such programs 
are still not achieving meaningful 
impacts on children and families 
in many countries. The two main 
reasons are lack of data due to the 
slow progress in information systems 
for monitoring and evaluation, and 
challenges in maintaining fidelity 
and quality at scale. These issues are 
well aligned with those presented in 
this important report. 

Latin-American countries have 
implemented two types of programs 
for very young children at scale: 
childcare services and, more recent- 
ly, home visits. For such programs to 
be successful a continuous quality 
improvement system has to be in 
place. But such system ought to be 
“nourished” by frequent data which 
is sometimes nonexistent in LAC. The 
data should, in turn, be used to im- 
prove different operational aspects 
of the programs that I discuss below. 

There is very little data on 
meaningful measures of quality of 
childcare services that could help 


set and monitor quality standards. 
However, two recent efforts of 
data collection illuminate the 
debate on the impact of daycares 
on child development and show 
a rather poor overall quality of 
the service provided. Bernal and 
Fernandez (2013) used the FDCRS 
(Family Day Care Rating Scale) 
that uses values from 1 (minimum 
quality) to 7 (optimal quality) to 
assess the quality of the public 
community-based daycares Hogares 
Comunitarios de Bienestar (HC) 
in Colombia. They found very 
unsatisfactory quality scores 
(average score of 2.3, with a 2.1 
in the process quality sub-scale) 
of this program that serves about 
800,000 children^ Similarly, Araujo, 
Lopez Boo, Novella, Schodt, and 
RominaTome (2013) applied the 
ITERS-R (Infant/Toddler Environment 
Rating Scale, Revised edition) and 
Toddler CLASS (Toddler Classroom 
Observation Scoring System) in the 
Ecuadorian public daycares Centres 
Infancies del Buen Vivir. Both 
instruments presented low average 
scores. For instance, the CLASS 
that focuses on measuring process 
quality presented a score of about 2 
in the instructional support domain 


and 4 in the emotional support and 
classroom organization domains out 
of a total of 7 points. 

The ITERS and ECERS have 
also been applied in Brazil, Peru 
and Chile and the CLASS has been 
applied in Chile and Peru. While 
the results vary across countries, 
and across sites within countries, 
evidence points to two key facts: 

(1 ) quality is often very low, 
especially the quality of care 
received by the most vulnerable 
children, and (2) when there is 
any measurement of quality by the 
programs themselves, it generally 
focuses on easily measurable inputs 
and infrastructure, rather than 
on processes (in particular, the 
quality of the interactions between 
caregivers and children). On the 
other hand, data on the quality of 
home visits has not been collected 
yet in at scale programs, but some 
efforts are underway. 

What are the challenges 
of implementation behind these 
numbers? Recurring themes in 
terms of quality of childcare are: 
deficiencies in the curriculum, 
insufficient initial training and on- 
the-job coaching, poor supervision 
and professional development 



practices, and almost nonexistent 
data monitoring systems to provide 
child-level information. On the other 
hand, for home visiting programs, 
supervision and monitoring, and the 
lack of data (e.g., inputs, products, 
outcomes) to monitor progress are 
major impediments to fidelity of 
implementation. Fitting evidence 
to the local context is very difficult 
when there is no skilled human 
capital to implement or supervise 
the home visits. There are also 
important challenges in recruitment 
and retention of staff with the 
required characteristics for the 
program. 

Overall, the challenges of 
going to scale in LAC are associated 
with: the complex and expensive 
task of monitoring and appropriate 
supervision and coaching of 
personnel, regional heterogeneity, 
scarce human resources, and 
insufficient investment in pertinent 
training. Scalability with quality in 
ECD seems to be facilitated by using 
existing services and staff (i.e., 
using health staff, teachers) and 
adding key factors such as training 
and coaching, while not requiring 
too much time from the parents 
to avoid drop out. If meaningful 
impacts on children and families are 
to be achieved in Latin-American 
countries, evidence-based programs 
will need to be strongly supported 
for quality implementation. As the 
report rightly states “the dialogue 
tying data monitoring and quality 
implementation can improve the 
sustainability of evidence in practice 
settings”. 
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